Automated Phrase Mining from Massive Text Corpora

As one of the fundamental tasks in text analysis, phrase mining aims at extracting quality phrases from a text corpus. Phrase mining is important in various tasks including automatic term recognition, document indexing, keyphrase extraction, and topic modeling. Most existing methods rely on complex, trained linguistic analyzers, and thus likely have unsatisfactory performance on text corpora of new domains and genres without extra but expensive adaption. Recently, a few data-driven methods have been developed successfully for extraction of phrases from massive domain-specific text. However, none of the state-of-the-art models is fully automated because they require human experts for designing rules or labeling phrases. In this paper, we propose a novel framework for automated phrase mining, AutoPhrase, which can achieve high performance with minimal human effort. Two new techniques have been developed: (1) by leveraging knowledge bases, a robust positive-only distant training method can avoid extra human labeling effort; and (2) when the part-of-speech (POS) tagger is available, a POS-guided phrasal segmentation model can better understand the syntactic information for the particular language and further enhance the performance by considering the context. Note that, AutoPhrase can support any language as long as a general knowledge base (e.g., Wikipedia) in that language are available, while benefitting from, but not requiring, a POS tagger. Compared to the state-of-the-art methods, the new method has shown significant improvements on effectiveness on five real-world datasets in different domains and languages.